Facial Recognition

Stephanie Kobakian

Mitchell O’Hara-Wild

26 September 2016

Winter research project

Project goals

  • Find the best facial recognition software for tennis matches
  • Build manually classified dataset of faces
  • Compare software performance under varied conditions
  • Determine ideal camera feeds for facial recognition

Challenges

  • Access to usable images and videos
  • Availability of software
  • Using the software
  • Time constraints

Applications of facial detection software

Camera auto-focus

Camera auto-focus

Facebook’s image tagging

Facebook’s tagging feature for quick classification of friends

Snapchat filters

snap

Surveillance and security

security

Google’s Mobile Vision

Google Mobile Vision

Applying facial recognition to tennis

Emotion in tennis

Serena Williams

“Tennis is mostly mental. You win or lose the match before you even go out there.”

Serena GIF

Long-term goals

  • Better understand emotion’s effect on player performance
  • Automatic collection of player emotion data from video

Manual classification

Dataset

  • 6406 Australian Open images (2.8GB)
  • 800x450px size frames from 105 match broadcast videos
  • Video frames taken every 3 seconds over a 5 minute segment

So how do we classify these?

  • Do it in Excel
  • Write a website
  • Write a program

Shiny Application

ShinyApp

Facial recognition software

Evaluated software (APIs)

  • Animetrics
  • Skybiometry
  • Microsoft
  • Google

Comparison of features

Feature Google Microsoft Skybiometry Animetrics
Face features ✔ ✔ ✔ ✔
Head angle ✔ ✔ ✔ ✔
Basic emotion ✔ ✔
Face landmarks ✔ ✔ ✔
Scene labels ✔
3D points ✔

Using the software

API: Application programming interface

APIs are built to be part of a larger program.

(for example, a camera app)

In order to use them, we need to write scripts to use the API.

Running the scripts

Detecting faces in each image typically took 3-4 seconds.

To complete all 6406 images, it would take 5-7 hours, per API.

Some APIs had usage limits, and took several days to run.

Raspberry Pie

RaspPie

Raspberry Pi

RPi

Raspberry Pi

RaspPi

Several days later…

We have all our data!
(toward the end of the original project window)

Preparing the data

Overlaying face boxes

Check if the software is producing sensible output.

Data is reasonable!

Most of the time.

Merging the data

Collected data was spread across 6 files, to analyse it, we need to merge it all together.

Data structure

Matching face boxes

We also need to determine if the boxes from each API match the same face.

This is also important for importing the extra face information from the manual classification.

Comparing every box generated, if the intersection of the box was >10%, they matched.

box intersection

Matching face boxes

Analysing the data

Shot angles

Best shot angle

Player close-up closeup

Worst shot angle

Birds-eye view birdseye

Hats and visors

As expected, they reduce the detection.

Classification time

time

Face size

size

Conclusions

The best facial recognition software

Google

  • Provided the most accurate face detection
  • Provided lots of meta information
  • Includes basic emotion recognition
  • Well priced

Room for improvement

Facial recognition software needs work to improve detection of faces, let alone emotion.

Some basic emotion is possible, however the precision is questionable, especially with players expressions during a game.

Further research

  • Use higher definition images
  • Obtain and use only best camera angles
  • Analyse player microphone audio for emotion
  • Use videos for facial recognition

Q & A